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ABSTRACT 

In  this  study,  we  simulated  a  generic  mounted  crewstation 
environment  and  conducted  an  experiment  to  examine  the 
workload  and  performance  of  the  operator  of  a  ground 
robot.  Participants  were  randomly  assigned  to  four  tasking 
conditions:  robotics  tasks  only,  robotics  plus  an  auditory 
task,  robotics  plus  a  visual  monitoring  task,  or  all  three 
tasks  simultaneously.  Participants  completed  four  mission 
scenarios.  In  two  of  these  scenarios,  their  robot  was  semi- 
autonomous.  In  the  other  two  scenarios,  they  had  to 
teleoperate  the  robot.  An  Aided  Target  Recognition 
(AiTR)  system  was  available  to  help  them  with  their 
target  detection  tasks  in  only  two  of  the  four  scenarios. 
Results  showed  that  operators’  situational  awareness  and 
perceived  workload  were  significantly  worse  when  they 
teleoperated  the  robot.  Individual  differences  factors  such 
as  the  operator’s  spatial  ability  and  attentional  control 
were  also  investigated.  Implications  for  military  personnel 
selection  were  discussed. 

1.  INTRODUCTION 

A  mandate  passed  by  the  United  States  Congress  in  2002 
called  for  1/3  of  all  Army  systems  to  be  unmanned  by 
2015.  This  mandate  included  an  assessment  which 
covered  four  classes  of  vehicles:  1)  reconnaissance, 
monitoring,  and  recovery  operations;  2)  transportation  of 
supplies  and  equipment;  3)  an  armed  vehicle  to  augment 
manned  systems;  and  4)  a  fully  autonomous  combat 
vehicle  (National  Research  Council,  2003).  The 
introduction  of  unmanned  systems  (i.e.,  robots)  to  the 
battlefield  has  obvious  advantages  (extend  manned 
capabilities,  act  as  force  multipliers,  and  most  importantly 
save  lives)  but  will  also  create  unique  challenges. 
Operators  in  complicated  environments  will  have  to 
handle  a  new  variety  of  complex  issues  posed  by  the  use 
of  unmanned  systems  (Jentsch  et  al.,  2004).  Often,  while 
utilizing  technology,  operators  are  required  to  assess  the 
current  state  of  the  given  system  and  operating 
environment  while  quickly  making  decisions  and 
executing  an  appropriate  course  of  action.  While  the 
utilization  of  a  suitable  technology,  such  as  unmanned 


ground  vehilces  (UGVs),  can  increase  the  likelihood  of 
successful  operations,  operator  performance  can  be 
adversely  impacted  by  the  task  load  associated  with  use  of 
the  technology. 

Future  unmanned  system  operators  are  likely  to  require 
some  level  of  automation  to  complete  complex  robot 
missions.  Appropriately  applied,  automation  can  enable 
operators  to  perform  at  greater  capacity.  The  key  will  be 
to  apply  automation  to  tasks  that  require  assistance  or  by 
automating  simple  tasks  to  free  up  operator  resources  and 
allow  them  to  attend  to  more  challenging  or  cognitively 
demanding  tasks.  Further,  automation  must  be  applied 
within  the  appropriate  context  or  time.  Through  the 
integration  of  physiological  measurement,  it  should  be 
possible  to  identify  the  appropriate  context  or  time  to 
apply  automation.  The  resultant  automation  process 
would  create  a  system  capable  of  detecting  and  utilizing 
the  state  of  the  operator  resulting  in  a  truly  ‘human-in-the- 
loop’  system.  This  current  effort  examines  the  impact  of 
automation  on  a  UGV  operator’s  performance  in  a 
simulated  military  multi-tasking  environment.  The  goal  is 
to  identify  the  high  workload  peaks  and  performance 
decrements  during  a  robot  mission  through  objective 
physiological  means. 

2.  METHOD 

2.1  Participants 

A  total  of  sixty-four  college  students  from  the  University 
of  Central  Florida  and  the  United  States  Military 
Academy  (25  female,  39  male)  participated  in  this  study. 
The  participants  ranged  in  age  from  18  to  49  (M  =  20.42, 
SD  =4.51).  The  experiment  lasted  approximately  3.5 
hours  and  volunteers  received  compensation  in  the  form 
of  extra  course  credit  or  $35.00  for  their  participation. 

2.2  Apparatus 
2.2.1  Simulation 

The  simulated  robotic  task  environment  was  provided  by 
the  Mixed  Initiative  experimental  (MIX)  Testbed.  A 
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detailed  description  of  the  design,  implementation,  and 
capabilities  of  the  MIX  Testbed  is  provided  in  Barber, 
Davis,  Nicholson,  Finkelstein,  and  Chen  (in  press)  and 
Barber,  Leontyev,  Sun,  Davis,  and  Nicholson  (2006). 

Robotic  Vehicle  Control.  One  of  the  main  components  of 
the  MIX  Testbed  is  the  Unmanned  Vehicle  Simulator 
(UVSIM),  which  generates  the  robotic  systems  in  the 
simulation.  For  this  study,  an  UGV  was  selected  as  the 
simulated  robotic  vehicle.  The  UGV  provides  manual 
control  and  two  types  of  vehicle  automation,  waypoint 
navigation  (i.e.,  semi-autonomous  control)  and  aided 
target  recognition  (AiTR)  (Figure  la).  In  conditions 
requiring  manual  navigation  of  the  vehicle 
(teleoperation), throttle,  steering  and  direction  (forward  or 
reverse)  was  controlled  through  the  use  of  a  standard 
joystick.  In  contrast,  the  vehicle  automatically  followed  a 
pre-planned  path  (i.e.,  mission  route)  and  paused  at  four 
designated  checkpoints  in  the  semi-autonomous 
conditions.  In  conditions  utilizing  the  AiTR  capability, 
the  vehicle  automatically  scanned  for  targets,  by  panning 
the  vehicle’s  camera,  and  provided  a  list  of  all  targets 
(enemy  and  friendly)  in  the  area,  whereas  the  manual 
control  conditions  required  participants  to  teleoperate  the 
vehicle  in  the  designated  checkpoint  area  in  order  to 
properly  position  the  vehicle  for  target  observation.  In  the 
current  study,  the  two  types  of  vehicle  automation  were 
combined  with  the  associated  manual  control  conditions 
to  produce  four  control  conditions:  teleoperation  with  and 
without  AiTR  and  semi-autonomous  with  and  without 
AiTR. 

Operator  Control  Unit.  Another  main  component  of  the 
MIX  Testbed  is  the  Operator  Control  Unit  (OCU),  which 
is  the  graphical  user  interface  that  enabled  participants  to 
interact  with  the  unmanned  vehicle  (see  Figure  lb  and  c). 

The  OCU  also  provided  two  additional  tasks,  which  were 
used  to  vary  the  participants’  task  load.  The  visual 
monitoring  task  displayed  four  gauges  that  were 
constantly  in  motion  (shown  in  Figures  lb  and  lc-  four 
green  gauges  at  the  bottom).  Based  on  pre-specified 
timings  contained  in  a  settings  file,  one  or  more  of  the 
gauges  would  enter  an  upper  or  lower  limit  at  various 
times  throughout  each  of  the  missions.  The  gauge(s) 
would  return  to  normal  levels  when  a  “Reset”  button  was 
pressed.  The  auditory  communications  task  presented  pre¬ 
recorded  audio  cues  (i.e.,  a  call  sign)  at  various  times 
throughout  each  scenario.  Participants  used  a  keyboard  to 
enter  a  response  to  the  audio  cue  into  the  communications 
panel  located  beneath  the  visual  monitoring  task  on  the 
OCU.  Four  pre-recorded  questions,  which  were  designed 
to  assess  participants’  situation  awareness  (SA),  were  also 
presented  by  the  OCU  when  the  unmanned  vehicle 
entered  a  pre-specified  region  of  the  map.  Participants 
used  the  keyboard  to  enter  a  text  response  to  the  SA 
probes. 


The  OCU  also  provided  the  ability  to  automatically  time 
stamp  and  log  the  data  for  various  simulation  events  (e.g., 
audio  cue  presentation),  user  interactions  (e.g.,  keyboard 
input),  and  physiological  sensors  data. 


Figure  la  -  Aided  target  recognition  scan 


Figure  lb  -  Operator  control  unit  (map  view) 


Figure  lc  -  Operator  control  unit  (video  view) 


2.2.2  Wearable  Arousal  Meter 

The  wearable  arousal  meter  (WAM)  is  a  portable,  non- 
intrusive  device  that  measures  interbeat  heart  rate  interval. 
The  WAM’s  associated  software  provides  an  assessment 


of  the  wearer’s  state  of  arousal  derived  from  the  high 
frequency  values  of  the  interbeat  interval  (Hoover  & 
Muth,  2004).  (see  Figure  2).  Continuous  data  sampling 
was  provided  by  three  electrodes  (two  for  active 
recording,  one  for  signal  noise  reduction)  that  were 
connected  to  the  WAM  and  attached  by  the  participant  to 
their  sternum,  first  rib  on  the  right  side  of  the  torso,  and 
first  rib  on  the  left  side  of  the  torso. 


Figure  2  -  Wearable  arousal  meter 


2.2.3  Spatial  Tests  and  Questionnaires 

A  questionnaire  about  attentional  control  (Derryberry  & 
Reed,  2002)  was  used  to  evaluate  participants’  perceived 
attentional  control  (PAC).  The  attentional  control  survey 
consists  of  21  items  and  measures  perceived  attention 
focus  and  shifting.  The  scale  has  been  shown  to  have 
good  internal  reliability  (a  =  .88).  Derryberry  and  Reed 
conducted  an  experiment  to  examine  the  relationship 
between  self-reported  (i.e.,  attentional  control  survey 
score)  and  actual  attentional  control.  They  found  that 
participants  with  a  high  survey  score  could  better  resist 
interference  in  a  Stroop-like  spatial  conflict  task.  In  our 
previous  studies  (Chen  &  Joyner,  2006),  we  observed  a 
positive,  although  somewhat  weak,  relationship  between 
attentional  control  survey  score  and  some  multitasking 
performance  measures. 

The  Cube  Comparison  and  Hidden  Patterns  tests 
(Ekstrom  et  al.,  1976)  as  well  as  the  Spatial  Orientation 
test  were  used  to  assess  participants’  spatial  ability  (SpA). 
The  Cube  Comparison  test  requires  participants  to 
compare,  in  3-minutes,  21  pairs  of  6-sided  cubes  and 
determine  if  the  rotated  cubes  are  the  same  or  different. 
The  Hidden  Patterns  test  measures  flexibility  of  closure 
and  involves  identifying  specific  patterns  or  shapes 
embedded  within  distracting  information.  The  Guilford- 
Zimmerman  Spatial  Orientation  test  (Guilford  & 
Zimmerman,  1948)  provides  a  measure  of  a  person’s 
ability  to  identify  changes  in  direction  and  position.  The 
paper-and-pencil  test  consists  of  60  forced-choice  test 
items;  for  each  question,  the  change  in  a  motorboat’s 
heading  in  one  image  must  be  determined  from  the 
original  position  in  a  reference  image. 


Participants’  perceived  workload  was  evaluated  with  the 
computer-based  version  of  National  Aeronautics  and 
Space  Administration-task  load  index  (NASA-TLX) 
questionnaire  (Hart  &  Staveland,  1988).  The  NASA-TLX 
is  a  self-reported  questionnaire  of  perceived  demands  in 
six  areas:  mental,  physical,  temporal,  effort  (mental  and 
physical),  frustration,  and  performance.  Participants  were 
asked  to  evaluate  their  perceived  workload  level  in  these 
areas  on  10-point  scales.  They  also  assessed  the 
contribution  (i.e.,  weight)  of  each  factor  to  the  perceived 
workload  by  comparing  the  15  possible  pairs  of  the  six 
factors.  According  to  Noyes  and  Bruneau  (2007), 
computer-based  NASA-TLX  tends  to  generate  higher 
workload  ratings  compared  with  the  traditional  paper- 
based  survey.  However,  since  the  ratings  were  used  to 
compare  the  workload  levels  across  the  experimental 
conditions,  the  elevated  ratings  should  not  affect  these 
comparisons. 

2.2.4  Experimental  Design 

The  overall  design  of  the  study  isa2x2x2x2  mixed 
design.  The  between-subject  variables  were  Taskload  - 
Audio  (With  Audio  task  vs.  No  Audio  task)  and  Taskload 
-  Visual  (With  Visual  task  vs.  No  Visual  task). 
Participants  received  one  of  the  following  four  levels  of 
task  load:  (1)  robotic  tasks  only;  (2)  robotic  tasks  and 
visual  monitoring;  (3)  robotic  tasks  and  auditory 
monitoring;  and  (4)  robotic  tasks,  visual  monitoring,  and 
auditory  monitoring.  The  two  within- subjects  variables 
were  the  level  of  autonomy  for  the  robotic  vehicle 
navigation  (Robotics  Task  condition:  Semi- Autonomous 
vs.  Teleop)  and  the  level  of  aiding  for  target  identification 
(AiTR  condition:  With  AiTR  vs.  No  AiTR)  in  a 
preplanned  scan  area.  In  order  to  examine  these  two 
within- subjects  variables,  the  two  types  of  automation 
were  factorially  combined  to  produce  four  mission 
scenarios,  each  lasting  a  maximum  of  20  minutes.  Thus, 
the  levels  of  automation  included:  1)  semi-autonomous 
mode  with  AiTR;  2)  semi-autonomous  mode  without 
AiTR;  3)  teleoperation  with  AiTR;  and  4)  teleoperation 
without  AiTR. 

2.2.5  Procedure 

Upon  arrival  to  the  laboratory,  participants  were  assigned 
to  one  of  the  four  experimental  groups.  After  receiving  a 
briefing  about  the  purpose  of  the  study,  the  required  tasks 
and  any  risks  associated  with  participation,  all  volunteers 
were  given  an  informed  consent  form  to  read  and  sign. 
Participants  then  were  asked  to  complete  the 
demographics  form  and  the  attentional  control  survey 
followed  by  the  three  spatial  ability  tests  (Cube 
Comparison,  Hidden  Patterns,  and  Spatial  Orientation). 

After  completing  the  spatial  ability  tests,  participants  were 
instructed  on  the  proper  placement  of  the  three  WAM 
electrodes  and  asked  to  affix  the  WAM  electrodes  to  their 


torso;  assistance  was  provided  if  required.  The  WAM 
calibration  process  was  then  initiated.  Participants  were 
asked  to  sit  quietly  with  their  eyes  closed  for  10  minutes 
in  order  to  provide  a  resting  baseline  for  the  WAM.  After 
the  10  minute  period,  participants  received  a  three-phase 
training  session  on  the  proper  operation  of  the  UGV  and 
the  specific  tasks  they  would  need  to  accomplish  during 
each  of  the  four  experimental  missions. 

In  the  first  phase  of  training,  participants  viewed  a  slide 
show  that  demonstrated  the  proper  operation  of  the  UGV. 
During  the  slideshow,  participants  were  able  to  interact 
with  the  UGV  in  the  MIX  Testbed  for  familiarization. 
During  the  second  training  phase,  participants  completed 
a  test  mission  and  were  encouraged  to  ask  for  clarification 
on  any  tasks  or  operations  they  did  not  understand.  If  a 
participant  became  “lost”  and  did  not  ask  for  help,  the 
experimenter  provided  unsolicited  instruction.  The  final 
phase  of  training  required  the  participant  to  complete  a 
practice  mission  without  assistance.  If  the  participant 
needed  assistance,  they  returned  to  the  second  phase  until 
they  were  able  to  demonstrate  the  ability  to  complete  the 
practice  mission  without  assistance.  Once  the  training 
session  was  complete  (approximately  45  minutes), 
calibration  of  the  WAM  was  completed  by  retrieving 
participant  specific  baseline  data  from  the  WAM’s 
accompanying  software  and  initializing  the  WAM  to  the 
participant. 

Once  the  WAM  was  calibrated,  the  participant  began  the 
first  of  four  experimental  missions.  The  test  missions 
were  presented  in  the  same  order  for  each  participant 
allowing  participants  to  complete  a  mission  with  each 
level  of  autonomy.  The  order  of  presentation  for  the  level 
of  autonomy  was  counterbalanced  for  the  experimental 
missions. 

Each  mission  was  conducted  in  the  same  urban  terrain  but 
differed  in  the  pre-planned  route  and  placement  of 
specific  checkpoints,  non-targets  (civilians),  and  targets 
(dismounted  infantry  soldiers).  Each  route  was  comprised 
of  a  start  point,  an  end  point,  and  four  checkpoints. 
Participants  were  instructed  to  follow  the  route  to  each 
checkpoint.  While  traveling  between  checkpoints, 
participants  in  all  conditions  were  required  to  identify 
four  targets,  which  were  encountered  along  the  route  and 
outside  of  a  designated  scan  area,  by  pressing  the  “Threat 
Detected”  button.  Additionally,  when  a  participant  arrived 
at  each  of  the  checkpoints,  they  were  required  to  identify 
targets  in  a  designated  scan  area.  Once  a  target  was 
identified,  participants  were  required  to  plot  the  location 
of  each  target  on  their  virtual  terrain  map  and  complete  a 
spot  report  describing  the  target.  While  traversing 
between  checkpoints,  participants  were  also  asked  four 
prerecorded  situation  awareness  probes.  When 
participants  received  a  probe,  the  simulation  would  pause, 
the  screen  would  black  out,  and  participants  were  allowed 
12  seconds  to  type  their  answer  into  the  designated  text 


box  and  press  the  ’’Send”  button.  Once  the  timer  expired, 
the  blackout  was  removed  and  the  mission  continued 
exactly  as  it  was  prior  to  the  situation  awareness  probe. 

In  addition  to  the  basic  mission  tasks  common  to  all 
conditions  described  above,  those  participants  in  the 
visual  monitoring  task  condition  were  required  to  monitor 
the  gauges  that  indicated  the  “health”  of  the  UGV 
batteries  (the  performance  of  the  simulation  was  not 
affected).  If  one  or  more  of  the  gauges  moved  to  the 
critical  zone,  the  participant  was  required  to  press  the 
“Reset”  button.  Participants  in  the  audio  monitoring  task 
condition  were  required  to  monitor  a  continuous  series  of 
communications  for  their  designated  call  sign,  determine 
if  they  had  heard  their  call  sign,  and  enter  ‘Y’  or  ‘N’  in 
the  Communications  panel,  and  press  the  ‘“Send”  button 
in  response  to  each  communication.  Participants  who 
were  assigned  to  the  visual  and  auditory  monitoring  task 
condition  were  required  to  perform  both  tasks  while 
executing  the  common  mission  tasks.  Immediately 
following  the  conclusion  of  each  mission,  participants 
completed  the  NASA-TLX. 

Once  the  final  NASA-TLX  was  completed  after  the  fourth 
mission,  participants  completed  the  Simulator  Sickness 
Questionnaire  (SSQ)  to  ensure  they  were  not  experiencing 
any  symptoms  of  simulator  sickness  as  a  result  of 
exposure  to  the  simulated  environment.  Participants  then 
removed  the  WAM  electrodes  and  were  debriefed  and 
thanked  for  their  participation  in  the  study. 

2.2.6  Dependent  Measures 

The  dependent  variables  measured  during  the  study 
included:  target  identification  performance  (i.e.,  the 
proportion  of  targets  identified  at  the  checkpoints),  threat 
detection  performance  (i.e.,  the  proportion  of  correct 
enemy  threats  detected  outside  the  designated  target  scan 
areas),  performance  on  the  situation  awareness  probes, 
perceived  workload,  and  average  physiological  arousal 
level. 

3.  RESULTS 

3.1  Operator  Performance  -  Targets  Identified  at  the 
Checkpoints 

Table  1  lists  the  percentages  of  targets  identified  at  the 
checkpoints.  A  mixed  analysis  of  covariance  (ANCOVA) 
was  performed  to  examine  the  participants’  target 
detection  performance  (percentage  of  targets  detected  and 
identified  at  the  checkpoints),  in  the  AiTR  condition  (with 
AiTR  vs.  no  AiTR)  and  the  Robotics  Task  condition 
(Semi- Auto  vs.  Teleop)  being  the  within- subject  factors, 
the  Taskload  conditions,  Audio  (Audio  task  vs.  No  Audio 
task)  and  Visual  (Visual  task  vs.  No  Visual  task)  as  the 
between- subject  factors,  and  participants’  SpA  and  PAC 
as  co variates.  None  of  the  main  effects  were  statistically 
significant.  There  was  a  significant  AiTR  x  Audio 


interaction,  F(  1,  57)  =  4.699,  p  <  .05.  Figure  3  shows  the 
effects  of  Robotics  and  AiTR,  and  Figure  4  shows  the 
interaction  between  AiTR  and  Audio.  Participants’ 
performance  appeared  to  be  worse  when  they  had  to 
simultaneously  perform  the  Audio  task  without  the  aid  of 
AiTR  compared  to  the  With- AiTR  condition. 

3.2  Operator  Performance  -  Threats  Detected  along 
the  Route 

A  mixed  ANCOVA  was  performed  to  examine  the 
participants’  threat  detection  performance  (percentage  of 
threats  detected  along  the  route),  with  the  AiTR  condition 
and  the  Robotics  Task  condition  being  the  within- subject 
factors,  the  Taskload  conditions  as  the  between-subject 
factors,  and  participants’  SpA  and  PAC  as  covariates. 
There  was  a  significant  Robotics  x  PAC  interaction,  F(  1, 
57)  =  10.638,  p  <  .05.  Figure  5  shows  the  interaction 
between  Robotics  and  PAC.  Participants  with  high  PAC 
performed  better  with  the  Teleop,  while  low  PAC 
participants  performed  better  with  the  Semi-auto  robot. 
There  was  also  a  significant  Robotics  x  AiTR  x  SpA 
interaction,  F(  1,  57)  =  4.616, p  <  .05. 


Table  1.  Operator  Task  Performance  (mean  percent  of 
targets  detected  at  the  checkpoints) 


Semi- Auto 

Teleop 

AiTR 

No  AiTR 

AiTR 

No  AiTR 

Visual 

.976 

.858 

.881 

.893 

Audio 

No 

Visual 

.836 

.778 

.918 

.764 

No 

Audio 

Visual 

.884 

.910 

.860 

.872 

No 

Visual 

.884 

.860 

.820 

.926 

Figure  3  -  Targets  identified  at  checkpoints. 


Percentage  of  Targets  Identified  (at  checkpoints) 


Figure  4  -  Targets  identified  at  checkpoints  - 
interaction  between  AiTR  and  Audio  task. 

Threats  Detected  (along  the  route)-  Interaction  between  Robotics  and  PAC 


Figure  5  -  Threats  detected  along  the  route  - 
interaction  between  Robotics  and  PAC. 

3.3  Operator  Situational  Awareness 

A  mixed  ANCOVA  was  performed  to  examine  the 
participants’  SA  (composite  score  of  SA  Level  1,  2,  &  3 
queries),  with  the  AiTR  condition  and  the  Robotics  Task 
condition  being  the  within- subject  factors,  the  Taskload 
conditions  as  the  between-subject  factors,  and 
participants’  SpA  and  PAC  as  co variates.  The  analysis 
revealed  a  significant  main  effect  for  Robotics,  F(  1,  57)  = 
5.485,  p  <  .05  and  SpA,  F(  1,  57)  =  7.515,  p  <  .01. 
Participants’  SA  was  significantly  better  in  the  Semi- Auto 
conditions  than  in  the  Teleop  conditions.  Additionally, 
participants  with  higher  SpA  significantly  outperformed 
their  lower- SpA  counterparts.  There  was  a  significant 
Robotics  x  AiTR  x  SpA  interaction,  F(  1,  57)  =  5.242,/?  < 
.05.  Figure  6  shows  the  effects  of  Robotics  and  SpA. 


Situational  Awareness  (Composite) 


Perceived  Workload 


Figure  6  -  Situational  awareness  -  effects  of  Robotics 
tasks  and  operator  spatial  ability  (SpA). 

3.4  Operator  Perceived  Workload 

Table  2  lists  the  participants’  perceived  workload.  A 
mixed  ANCOVA  was  performed  to  examine  the 
participants’  perceived  workload  (weighted  NASA-TLX 
scores),  with  the  AiTR  condition  and  the  Robotics  Task 
condition  being  the  within- subject  factors,  the  Taskload 
conditions  as  the  between-subject  factors,  and 
participants’  SpA  and  PAC  as  covariates.  The  analysis 
revealed  a  significant  main  effect  for  Robotics,  F(l,  56)  = 
30.585,  p  <  .01  and  AiTR,  F(  1,  56)  =  6.765,  p  <  .05. 
Participants’  perceived  workload  was  significantly  higher 
in  the  Teleop  conditions  than  in  the  Semi- Auto 
conditions.  They  also  had  significantly  higher  workload 
when  there  was  no  AiTR,  compared  with  the  AiTR 
conditions.  There  was  a  significant  Robotics  x  AiTR 
interaction,  F(  1,  56)  =  5.053,  p  <  .05.  Figure  7  shows  the 
effects  of  Robotics  and  AiTR. 


Table  2.  Operator  Perceived  Workload 


Semi-Auto 

Teleop 

AiTR 

No  AiTR 

AiTR 

No  AiTR 

Visual 

48.7 

57.9 

60.9 

60.5 

Audio 

No 

Visual 

52.6 

57.4 

62.6 

65.5 

No 

Audio 

Visual 

46.5 

50.0 

53.5 

60.2 

No 

Visual 

44.7 

50.9 

54.3 

54.2 

Figure  7  -  Operator  perceived  workload  -  effects  of 
Robotics  and  AiTR. 

3.5  Operator  Physiological  Arousal  Level 

A  multivariate  analysis  of  variance  (MANOVA)  was 
performed  to  examine  the  participants’  physiological  state 
of  arousal  (average  arousal  level),  with  the  AiTR 
condition  and  the  Robotics  Task  condition  as  the  within- 
subject  factors,  the  Taskload  condition  as  the  between- 
subject  factor.  The  analysis  revealed  that  there  were  no 
significant  effects  of  levels  of  AiTR,  Robotics  Task,  or 
Taskload  (p  >  .10). 

4.  DISCUSSION 

In  this  study,  we  simulated  a  generic  mounted  crewstation 
environment  and  conducted  an  experiment  to  examine  the 
workload  and  performance  of  the  operator  of  a  ground 
robot.  Participants  were  randomly  assigned  to  four  tasking 
conditions:  robotics  tasks  only,  robotics  plus  an  auditory 
task,  robotics  plus  a  visual  monitoring  task,  or  all  three 
tasks  simultaneously.  Participants  completed  four  mission 
scenarios.  In  two  of  these  scenarios,  their  robot  was  semi- 
autonomous.  In  the  other  two  scenarios,  they  had  to 
teleoperate  the  robot.  An  Aided  Target  Recognition 
(AiTR)  system  was  available  to  help  them  with  their 
target  detection  tasks  in  only  two  of  the  four  scenarios.  It 
did  not  appear  that  any  of  the  main  factors  affected  the 
percentages  of  targets  that  participants  identified  at  the 
checkpoints.  However,  if  participants  had  to 
simultaneously  perform  the  Audio  task,  they  performed 
worse  when  they  did  not  have  access  to  the  AiTR 
compared  to  the  With- AiTR  condition.  In  contrast,  an 
opposite  trend  was  observed  for  the  No  Audio  task 
conditions.  It  is  likely  that  the  attentional  demand  from 
the  Audio  task  competed  against  the  RSTA  task. 
Therefore,  when  participants  had  to  concurrently  perform 
both  the  targeting  and  the  auditory  tasks  without  the  aid  of 


the  AiTR,  their  targeting  performance  degraded. 
However,  the  same  degradation  was  not  observed  for  the 
No- Audio  conditions. 

The  number  of  threats  that  participants  detected  along  the 
route  provided  an  estimate  of  the  participants’  visual 
attention  of  the  environment  while  they  were  conducting 
other  tasks.  The  results  showed  that  participants  with  high 
PAC  performed  better  with  the  Teleop,  while  low  PAC 
participants  performed  better  with  the  Semi- Auto  robot. 
These  findings  are  consistent  with  Chen  and  Terrence 
(2008b)  that  high  PAC  operators  tend  to  perform  better 
with  manually  operated  systems  than  do  low  PAC 
individuals  in  multi-tasking  environment. 

The  results  showed  that  participants’  SA  was  significantly 
better  in  the  Semi- Auto  conditions  than  in  the  Teleop 
conditions.  These  results  are  consistent  with  past  findings 
that  robotics  operators  demonstrated  higher  SA  when  the 
robot’s  level  of  automation  was  higher  (Chen,  Durlach, 
Sloan,  &  Bowens,  2008;  Dixon,  Wickens,  &  Chang, 
2003;  Luck,  Allender,  &  Russell,  2006).  These  findings 
suggest  that  the  attention  on  (manual)  robotics  control 
might  have  distracted  the  operators  from  paying  attention 
to  the  environment. 

Participants’  workload  assessment  was  found  to  be 
affected  by  the  type  of  robotics  task  as  well  as  whether 
their  RSTA  task  was  aided  by  AiTR.  They  reported  a 
higher  workload  level  when  their  RSTA  task  was 
unassisted  by  AiTR.  They  also  experienced  significantly 
higher  workload  when  they  teleoperated  the  robot.  These 
results  are  consistent  with  Chen  et  al.  (2008),  Chen  and 
Joyner  (2006),  Chen  and  Terrence  (2008a),  and  Schipani 
(2003),  who  evaluated  robotics  operator  workload  in  a 
field  setting.  Although  many  of  the  ground  robots  in  the 
U.S.  Army’s  Future  Combat  System  (FCS)  program  will 
be  semi-autonomous,  teleoperation  will  still  be  an 
important  part  of  any  missions  involving  robots  (e.g., 
when  robots  encounter  obstacles  or  other  situations 
requiring  operator  intervention).  Therefore,  the  higher 
workload  associated  with  teleoperation  needs  to  be  taken 
into  account  when  designing  the  user  interfaces  for  the 
robotic  systems.  For  potential  user  interface  design 
solutions,  see  Chen,  Haas,  and  Barnes  (2007). 

The  analyses  of  participants’  physiological  arousal  level 
failed  to  detect  a  significant  difference  between 
conditions.  One  potential  explanation  for  the  lack  of  the 
anticipated  effects  of  varying  task  load  levels  on  average 
arousal  may  be  due  to  the  nature  of  the  dependent 
variable.  That  is,  the  average  level  of  arousal  for  each 
condition  (i.e.,  mission)  does  not  appear  to  have  the 
sensitivity  required  to  capture  changes  in  arousal  that 
occurred  throughout  the  mission.  Consequently, 
additional  investigational  analyses  will  be  examined 
around  specific  points  in  the  mission  where  task  load  is 
likely  to  be  elevated. 


5.  FUTURE  RESEARCH 

Automation  will  likely  be  required  to  support  human- 
robot  performance,  but  when  automation  aides  should  be 
invoked  may  vary  based  on  individual  differences  of  the 
unmanned  system  operator  and  changes  in  the  task  load 
imposed  upon  them  throughout  a  mission.  Adaptive 
automation,  in  which  the  system  initiates  task  automation 
based  on  criteria  such  as  critical  events  or  operator 
performance,  has  been  proposed  as  an  effective  means  to 
enhance  human-robot  system  performance  (Parasuraman, 
Barnes,  &  Cosenzo,  2007).  Although  there  is  a  paucity  of 
empirical  investigations  on  the  efficiency  of  adaptive 
automation  for  unmanned  systems,  a  recent  study  of 
operator  performance  with  multiple  uninhabited  systems 
demonstrated  a  beneficial  effect  of  performance-based 
adaptive  automation  on  workload  and  performance 
(Parasuraman,  Cosenzo,  &  DeVisser,  in  press). 

In  our  upcoming  experiment,  which  is  designed  to  expand 
upon  the  Parasuraman  et  al.  (in  press),  the  differential 
effects  of  various  types  of  automation  on  operator 
performance,  workload,  and  situation  awareness  will  be 
empirically  evaluated.  Four  automation  conditions  will  be 
compared  during  a  simulated  high  workload 
reconnaissance,  surveillance,  and  target  acquisition 
mission:  manual,  static  automation,  and  two  methods  for 
invoking  adaptive  automation.  Specifically,  in  the 
performance-  and  physiological-based  adaptive 
automation  conditions,  real-time  assessment  of  operator 
performance  and  physiological  indicators  will  be  used  as 
triggers  for  task  automation. 
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